PKG-895: Allow citus and timescaledb to compile#600
Open
dutow wants to merge 17 commits into
Open
Conversation
The original part of the version is kept as is, and Percona specific information is added after. For example, psql displays the following after this commit: SELECT version(); PostgreSQL 17.6 (PostgreSQL) - Percona Server for PostgreSQL 17.6.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit Where the addition is the extra version at the end: " - Percona Server for PostgreSQL 17.4.1" with the ".1" being the Percona version number. Similarly, commands show the same information: bin/pg_ctl --version pg_ctl (PostgreSQL) 17.6 - Percona Server for PostgreSQL 17.6.1
There are various reasons why one would want to create their own implementation of a storage manager, among which are block-level compression, encryption and offloading to cold storage. This patch is a first patch that allows extensions to register their own SMgr. Note, however, that this SMgr is not yet used - only the first SMgr to register is used, and this is currently the md.c smgr. Future commits will include facilities to select an SMgr for each tablespace.
With this change, mdcreate receives the old relfilelocator along with the new for operations that create a new file for an existing relation. This is required for tde_heap in pg_tde.
and allow extensions to override it For now, it extends on `pread` and `pwrite` from/into segment files. This is the minimum we need for full XLog encryption with pg_de.
This commit modifies the build scripts so that `llvmjit.so` links LLVM statically instead of dynamically. No other binary affected, the change shouldn't be user visible at all. The memory footprint of postgres is also unaffected: only the `llvmjit.so` dynamic library is affected. If postgres doesn't use JIT, the LLVM library isn't loaded, as before. If it uses JIT, it is loaded as part of `llvmjit.so`, sa before. The only change is the binary size of `llvmjit.so`.
Instead of checking for the oid of the AM we check for if the access method of the relation uses the routine struct as the heap does. This allows the tools to be support all access methods which are simple copies of heap. I am uncertain if this is something which the project would be interested in but it decreases the size of our diff against upstream.
Need to modify the existing license to include the portions of code unique to Percona
The pg_tde extension's custom storage manager needs to add its own AIO callback to decrypt pages before the buffer manager's callback is called but since the set of AIO callbacks is fixed we need to expose this API so that extensions can register their own types of callbacks. This new function for registering a callback is supposed to be called when intializing shared memory. We try to keep the number of callbacks low, setting it to 15, so that we have the 4 built-in callbacks and 11 slots that extensions can use.
This commit reintroduces the old method signatures of some methods we changed to support pg_tde. The changed method signatures still exists with a "2" suffix, and all internal code calls the "2" methods. The reintroduced original methods also check a global flag which can disable them - if this happens, they report a FATAL error instead. This variable is activated by pg_tde, which means that if something tries to use these while pg_tde is loaded, and with that, possibly ignores its file tracking mechanism, it reports an error. In practice this means that timescaledb, or the columnar storage in citus won't work if pg_tde is loaded in the shared_preload_libraries, but they can be used with our distribution without pg_tde enabled. Also verified that both citus and timescaledb do compile with these changes, and their basic test suite can be run againts our fork.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Let's talk about this patch again!
Same as before, only for the PSP18 branch for now, 17 needs the same changes of course.
The pg_tde repo will need a matching
change if we go with this patch as-is.
This commit reintroduces the old method signatures of some methods we changed to support pg_tde. The changed method signatures still exists with a "2" suffix, and all internal code calls the "2" methods.
The reintroduced original methods also check a global flag which can disable them - if this happens, they report a FATAL error instead. This variable is activated by pg_tde, which means that if something tries to use these while pg_tde is loaded, and with that, possibly ignores its file tracking mechanism, it reports an error.
In practice this means that timescaledb, or the columnar storage in citus won't work if pg_tde is loaded in the shared_preload_libraries, but they can be used with our distribution without pg_tde enabled.
Also verified that both citus and timescaledb do compile with these changes, and their basic test suite can be run againts our fork.